Intraoperative Systems and Methods for Determining and Providing for Display a Virtual Image Overlaid onto a Visual Image of a Bone

ABSTRACT

Example methods and systems may be used intraoperatively to help surgeons perform accurate and replicable surgeries, such as knee arthroplasty surgeries. An example system combines real time measurement and tracking components with functionality to compile data collected by the hardware to register a bone of a patient, calculate an axis (e.g., mechanical axis) of the leg of the patient, assist a surgeon in placing cut guides, and verify a placement of an inserted prosthesis.

CROSS REFERENCE TO RELATED APPLICATION

The present disclosure is a continuation of and claims priority to U.S.patent application Ser. No. 16/260,251, filed on Jan. 29, 2019, which isa continuation of and claims priority to U.S. patent application Ser.No. 16/106,404, filed on Aug. 21, 2018, which is a continuation of andclaims priority to U.S. patent application Ser. No. 15/261,356, filed onSep. 9, 2016, which claims priority to U.S. patent application No.62/217,185, filed on Sep. 11, 2015, the entire contents of each of whichare herein incorporated by reference.

FIELD

The present disclosure relates generally to collecting information of abone of a limb of a patient and providing for display a virtual imageoverlaid onto a visual model of the bone, and more particularly torecognizing a position and a movement of the bone in physical space andrecognizing a position and a movement of a surgical device in physicalspace so as to provide information for assistance of placement of thesurgical device at a position on the bone based on the position and themovement of the surgical device in physical space.

BACKGROUND

During orthopedic knee arthroplasties, the worn out bone and cartilageare removed and replaced with various biocompatible implants to take theplace of the resected bone and cartilage. Knee arthroplasty usuallyincludes a femoral component that is fixed onto a distal end of a femur,and tibial components that include a tibial tray and an intermediatecomponent. To affix these components to the bone, a series of cuts aremade to the distal end of the femur and a proximal end of the tibia. Tomake these cuts in the bones, surgeons use a variety of cut guides thatare used to guide a blade of a bone saw through the bones of thepatient.

Many modern orthopedic knee arthroplasties use computer assistance tohelp the surgeon perform an accurate and replicable surgery. Currently,“navigation” systems are used for the computer to register the bone.These systems are able to help verify and estimate a general surface ofthe bone through a probing sequence that is timely and cumbersome forsurgeons. After the bone is registered, it is required during kneearthroplasties that mechanical axes of the leg are discovered to be ableto make the distal cut to the femur perpendicular to the axes. Manysystems use an intramedullary rod and/or an extramedullary rod, whileothers use contact infrared probes.

Therefore, there is a need for a system that is able to accuratelyregister the bone in real time, calculate the mechanical axes of the legwithout probes, and be able to guide the surgeon to accurately place theimplant/prosthesis through a real time measurement array.

SUMMARY

In one example, a system is described that includes a measurement deviceto collect information of a surface of a bone of a limb of a patient, totrack one or more reference points positioned on one or more bones or ananatomy of the patient, and to track one or more reference pointspositioned on a surgical device. The system also includes one or moreprocessors, and data storage including instructions executable by theone or more processors for performing functions. The functions comprisedetermining a visual model of the bone based on the collectedinformation of the surface of the bone, recognizing a position and amovement of the bone in physical space based on the tracked referencepoints positioned on the one or more of the bone or the anatomy of thepatient and causing the visual model of the bone to reflect the positionand the movement of the bone in physical space, recognizing a positionand a movement of the surgical device in physical space based on thetracked reference points positioned on the surgical device, andproviding for display a virtual image overlaid onto the visual model ofthe bone, wherein the virtual image includes information for assistingin placement of the surgical device at a position on the bone based onthe position and the movement of the surgical device in physical space.

In another example, a method is described comprising receivinginformation, collected by a measurement device, of a surface of a boneof a limb of a patient, determining a visual model of the bone based onthe collected information of the surface of the bone, recognizing aposition and a movement of the bone in physical space based on trackedreference points positioned on the one or more of the bone or theanatomy of the patient and causing the visual model of the bone toreflect the position and the movement of the bone in physical space,recognizing a position and a movement of a surgical device in physicalspace based on tracked reference points positioned on the surgicaldevice, and providing for display a virtual image overlaid onto thevisual model of the bone, wherein the virtual image includes informationfor assisting in placement of the surgical device at a position on thebone based on the position and the movement of the surgical device inphysical space.

In another example, a non-transitory computer-readable medium isdescribed having stored thereon instructions that when executed by acomputing device that includes one or more processors causes thecomputing device to perform functions. The functions comprise receivinginformation, collected by a measurement device, of a surface of a boneof a limb of a patient, determining a visual model of the bone based onthe collected information of the surface of the bone, recognizing aposition and a movement of the bone in physical space based on trackedreference points positioned on the one or more of the bone or theanatomy of the patient and causing the visual model of the bone toreflect the position and the movement of the bone in physical space,recognizing a position and a movement of a surgical device in physicalspace based on tracked reference points positioned on the surgicaldevice, and providing for display a virtual image overlaid onto thevisual model of the bone, wherein the virtual image includes informationfor assisting in placement of the surgical device at a position on thebone based on the position and the movement of the surgical device inphysical space.

The features, functions, and advantages that have been discussed can beachieved independently in various embodiments or may be combined in yetother embodiments further details of which can be seen with reference tothe following description and drawings.

BRIEF DESCRIPTION OF THE FIGURES

The novel features believed characteristic of the illustrativeembodiments are set forth in the appended claims. The illustrativeembodiments, however, as well as a preferred mode of use, furtherobjectives and descriptions thereof, will best be understood byreference to the following detailed description of an illustrativeembodiment of the present disclosure when read in conjunction with theaccompanying drawings, wherein:

FIG. 1 is a block diagram of a system, according to an exampleembodiment.

FIG. 2 is a representation of the system in an operating room, accordingto an example embodiment.

FIG. 3 illustrates an example visual model of a bone, according to anexample embodiment.

FIG. 4 illustrates a conceptual portion of a limb of a patient withreference points positioned on the bone for tracking, according to anexample embodiment.

FIG. 5 illustrates an example surgical device, according to an exampleembodiment.

FIG. 6 illustrates another view of the example surgical device,according to an example embodiment.

FIG. 7 is a conceptual illustration of a virtual image overlaid onto avisual model, according to an example embodiment.

FIG. 8 illustrates a conceptual portion of a limb of a patient as inFIG. 4 with a plane shown, according to an example embodiment.

FIG. 9 illustrates a conceptual portion of a limb of a patient with aplane shown, according to an example embodiment.

FIG. 10 illustrates a conceptual portion of a limb of a patient with amechanical axis shown, according to an example embodiment.

FIG. 11 illustrates a conceptual portion of the visual model of a limbof a patient with a prosthesis inserted, according to an exampleembodiment.

FIG. 12 illustrates a conceptual portion of the visual model of a limbof a patient with another prosthesis inserted, according to an exampleembodiment.

FIG. 13 illustrates a conceptual image of placement of a prosthesis andadditional data to be displayed, according to an example embodiment.

FIG. 14 shows a flowchart of an example method for determining andproviding for display a virtual image overlaid onto a visual image of abone, according to an example embodiment.

FIG. 15 shows a flowchart of another example method for positioning of aprosthesis, according to an example embodiment.

FIG. 16 shows a flowchart of another example method for positioning of aprosthesis, according to an example embodiment.

DETAILED DESCRIPTION

Disclosed embodiments will now be described more fully hereinafter withreference to the accompanying drawings, in which some, but not all ofthe disclosed embodiments are shown. Indeed, several differentembodiments may be described and should not be construed as limited tothe embodiments set forth herein. Rather, these embodiments aredescribed so that this disclosure will be thorough and complete and willfully convey the scope of the disclosure to those skilled in the art.

Example methods and systems described below may help surgeons make cutsto the bone during surgeries, such as knee arthroplasty replacementsurgeries. An example system can be used intraoperatively to register asurface of the bone to be cut in real time with a measurement device.The system will then track, with spatial awareness, the bone tocalculate mechanical axes of the leg. The system then assists thesurgeon in placing the cut guides in a correct position onto the bone.To do this, the system uses an array of measuring devices to providereal time data as well as object tracking when the surgeon moves the cutguides. In further examples, the system uses augmented reality with thereal time measurement array to assist the surgeon with placement of thecut guides by displaying virtual cutting planes onto a live image of thebone produced in reference to the physical cut guide before the cutguide is fixed into place.

Example modes of implementation are described below as one of many waysthat the embodiments may be carried out. The systems may includehardware and software computing components to enable a surgicalprocedure, along with physical components needed to guide the surgeonduring the procedure.

Referring now to FIG. 1, a block diagram of a system 100 is illustrated,according to an example embodiment. The system 100 includes ameasurement device(s) 102, a computing device 104, a display 118, andcamera(s) 120.

The measurement device(s) 102 may collect information of a surface of abone of a limb of a patient, track one or more reference pointspositioned on one or more bones or an anatomy of the patient, and trackone or more reference points positioned on a surgical device. Themeasurement device(s) 102 thus is used to register the bone, as well astrack movement of the bone and of surgical devices. As one example, themeasurement device(s) 102 may track different features of the anatomy ofthe patient as well as virtual reference points to aid in determiningmechanical axes of the leg, and track and measure spatially surgicaldevices such as cut guides, and also measure spatially a final placementof the prosthesis.

The measurement device(s) 102 may be, but is not limited to, anynon-contact active measurement device, non-contact passive measurementdevice, or non-contact three-dimensional (3D) scanning device. In otherexamples, the measurement device(s) 102 may include a high acquisitionrate laser scanner, a scanner attached to an articulating arm, or acorded mobile scanner. The measurement device(s) 102 may be operatedusing technologies such as time-of-flight, triangulation, conoscopicholography, structured light, modulated light, laser, infrared,stereoscopic 3D imaging, or any combination of the aforementionedtechnologies. One or more of these devices can be used in the system 100for a variety of differing functions. Thus, multiple measurement devicesmay be used.

In some examples, the measurement device is one or more cameras forcapturing live images and may be combined with the camera(s) 120, orreplaced by the camera(s) 120.

Examples of the measurement device(s) 102 can include a 3D laser scannerattached to a robotic arm, a portable coordinate measuring machine, or aportable handheld laser scanner.

The measurement device(s) 102 may be a wireless device or a wireddevice, such as a hand held device or attached by an articulating arm toobtain a view of the bone. In further examples, the measurementdevice(s) 102 may include an ultrasonic sensor to register the bonebefore an initial incision.

The measurement device(s) 102 collects information about the patient andsends the information to the computing device 104.

The computing device 104 includes processor(s) 106, and data storage 108including instructions 110 executable by the processor(s) 106 forperforming functions, an input interface 112, and an output interface114 each connected to a communication bus 116. The computing device 104may also include hardware to enable communication within the computingdevice 104 and between the computing device 104 and other devices (notshown). The hardware may include transmitters, receivers, and antennas,for example.

The processor(s) 106 may be a general-purpose processor or a specialpurpose processor (e.g., digital signal processors, application specificintegrated circuits, etc.). The processor(s) 106 may receive inputs fromthe data storage 108 and the input interface 112, and process the inputsto generate outputs that are stored in the data storage 108 and outputto the output interface 114. The processor(s) 106 can be configured toexecute the executable instructions 110 (e.g., computer-readable programinstructions) that are stored in the data storage 108 and are executableto provide the functionality of the system 100 described herein.

The data storage 108 may include or take the form of one or morecomputer-readable storage media that can be read or accessed by theprocessor(s) 106. The computer-readable storage media can includevolatile and/or non-volatile storage components, such as optical,magnetic, organic, semiconductor-based or other memory or disc storage,which can be integrated in whole or in part with the processor(s) 106.The data storage 108 is considered non-transitory computer readablemedia. In some embodiments, the data storage 108 can be implementedusing a single physical device (e.g., one optical, magnetic, organic orother memory or disc storage unit), while in other embodiments, the datastorage 108 can be implemented using two or more physical devices.

The data storage 108 thus is a non-transitory computer readable storagemedium, and executable instructions 110 are stored thereon. Theinstructions 110 include computer executable code. When the instructions110 are executed by the processor(s) 106, the processor(s) 106 arecaused to perform functions. Such functions include determining a visualmodel of a bone of a patient based on information collected by themeasurement device(s) 102 and/or the camera 120, recognizing a positionand a movement of the bone in physical space based on tracked referencepoints, recognizing a position and a movement of a surgical device inphysical space based on tracked reference points, and providing fordisplay a virtual image overlaid onto the visual model of the bone, forexample.

The input interface 112 may be a wireless interface and/or one or morewireline interfaces that allow for both short-range communication andlong-range communication to one or more networks or to one or moreremote devices. Such wireless interfaces may provide for communicationunder one or more wireless communication protocols, such as Bluetooth,WiFi (e.g., an institute of electrical and electronic engineers (IEEE)802.11 protocol), Long-Term Evolution (LTE), cellular communications,near-field communication (NFC), and/or other wireless communicationprotocols. Such wired interfaces may include data buses such as Ethernetinterface, a Universal Serial Bus (USB) interface, or similar interfaceto communicate via a wire, a twisted pair of wires, a coaxial cable, anoptical link, a fiber-optic link, or other physical connection to awireline network. Thus, the input interface 112 may be configured toreceive input data from the measurement device(s) 102 and the camera(s)120, and may also be configured to send output data to other devices.

The output interface 114 outputs information to the display 118 or toother components as well. Thus, the output interface 114 may be similarto the input interface 112 and can be a wireless interface (e.g.,transmitter) or a wired interface as well.

The display 118 may be a stand-alone display or incorporated within thecomputing device 104.

The camera(s) 120 may include a dedicated high resolution camera arrayfor real time tracking and augmented reality. As an example, thecamera(s) 120 may include two cameras or stereo cameras for 3D visionapplications. The cameras 120 will also be able to collect measurementinformation to track surgical objects. Further, the cameras 120 canobtain depth measurements and calculate distance measurements using astereo camera array. Within examples below, the camera(s) 120 collectimages and/or other data to track objects in real time. For example, thecamera(s) 120 may be used together with the measurement device(s) 102 totrack objects such as to acquire intraoperative data, or the camera(s)120 may perform the real time tracking of the bones, surgical devices,etc. themselves. The camera(s) 120 output image data to the inputinterface 112.

FIG. 2 is a representation of the system 100 in an operating room,according to an example embodiment. The measurement device 102 is shownas a handheld device to scan a bone of the patient. The computing device104 is shown as a traditional desktop computer with display, and thedisplay 118 is shown mounted on a standing mount such that the surgeoncan easily visualize the screen during surgery. The camera(s) 120 aremounted overhead to capture a birds-eye-view of the patient and surgery.

A surgeon may operate the measurement device(s) 102 as a handheld 3Dlaser scanner to pass the measurement device(s) 102 over the boneundergoing surgery and/or other anatomy of the patient to collectinformation and learn geometry of the bone with the laser. Theinformation may be point-cloud data about a surface of the bone of alimb of the patient, for example, that can be used by the processor(s)106 to determining a visual model of the bone.

The measurement device(s) 102 may be used during surgery so as tocollect the information of the surface of the bone as exposed from thelimb of the patient in real time during surgery, and the processor(s)perform functions to determine the visual model of the boneintraoperatively during surgery. Thus, surgery may begin and the bonecan be exposed, and following, a surface of the bone can be scanned bythe measurement device(s) 102 to create a visual model of the bone inreal time during surgery. As a specific example for a knee surgery, oncethe distal end of the femur and tibia are exposed to the greatest degreeduring surgery, the bone will be registered into the system 100.Registration of the bone can occur entirely intraoperatively to providethe most up to date and accurate data of the anatomy of the patient.

In other examples, the measurement device(s) 102 may be used prior tosurgery to collect pre-operative imaging of at least a portion of thelimb of the patient. In this example, pre-operative imaging can include,but is not limited to, CT scans, MRI, x-rays, etc., used to construct apreoperative 3D model. Once the pre-operative images are obtained, thepre-operative imaging can be aligned with the collected information ofthe surface of the bone obtained during surgery to further determine thevisual model of the bone based on the aligned pre-operative imaging withthe collected information of the surface of the bone. Geometry from thepre-operative imaging can be matched with geometry of actual anatomy ofthe patient by performing point cloud analysis and imaging matching.Intraoperative data can thus be used to align preoperative data toremove a process of using a touch probe to identify landmarks to alignpreoperative data. The processor(s) 106 can provide for display thepre-operative image on screen as well as the visual model created fromthe pre-operative imaging and information collected during surgery, andthe images move on screen with movement of the leg of the patient as aresult of optical tracking by the camera(s) 120.

FIG. 3 illustrates an example visual model 122 of a bone, according toan example embodiment. The visual model shows an outline of the bone in3D to illustrate features of the bone.

In some other examples, the visual model 122 includes a live image ofthe bone rather than an outline or graphics.

The measurement device(s) 102 and/or the camera(s) 120 also track one ormore reference points positioned on one or more bones or an anatomy ofthe patient, and the processor(s) 106 can recognize a position and amovement of the bone in physical space based on the tracked referencepoints positioned on the one or more of the bone or the anatomy of thepatient and cause the visual model of the bone to reflect the positionand the movement of the bone in physical space.

FIG. 4 illustrates a conceptual portion of a limb of a patient withreference points 124, 126, and 128 positioned on the bone for alignmentand/or tracking, according to an example embodiment. The referencepoints 124, 126, and 128 may be physical objects positioned on the boneat certain positions, or virtual reference points assigned to certainpositions on the bone that can be recognized in three dimensional space.The measurement device(s) 102 and/or the camera(s) 120 tracks thereference points 124, 126, and 128 by continuously taking measurementsduring surgery to identify movement of the bone, and providesinformation of movement of the bone to the processor(s) 106. Theprocessor(s) 106 can update a viewpoint of the visual model of the boneto reflect the position and the movement of the bone in physical space.In this way, the surgeon will have a visual model of the bone on thedisplay 118 that accurately reflects the position of the bone duringsurgery.

Thus, during surgery and after the exposed bone and leg is registeredinto the system 100, the surgeon may place the reference points 124,126, and 128 that will act as an array onto the bone that was justregistered, as well as on any other locations on the anatomy of thepatient as desired. The reference points 124, 126, and 128 may bevirtual reference points. In another example, physical reference pointscan be attached to the patient through adhesive properties, marking orcoloring the patient, pinning the reference points, and other techniquesas well. These points will act as an array and can be used by themeasurement device(s) 102 to recognize where the bone is in relation tothe visual model 122 for the registered bone. Then the data for theregistered bone will be referenced virtually to the actual patient'sbone in real time as the bone moves. The leg will be moved around invarious ways to track movement of this virtual or physical referencepoint(s). The measurement device(s) 102 and/or the camera(s) 120 can beused to track the motion of the virtual reference point(s) and thepatient's leg.

The measurement device(s) 102 and/or the camera(s) 120 can trackreference points positioned on any number of areas of the bone and/orthe anatomy of the patient, including on one or more of a distalanterior tibial crest of the patient, a medial malleoli of the patient,a lateral malleoli of the patient, a reference off a medial third orother portion of the tibial tubercle of the patient, a base of ananterior cruciate ligament of the patient, and a posterior cruciateligament and intercondylar eminences of the patient, for example. Inaddition, such reference points positioned in these areas can be used toidentify mechanical axes or align preoperative scan data, as discussedmore fully below.

In another example, to recognize the position and the movement of thebone in physical space, tracked reference points positioned on apositioning device can be determined. For example, a portion of the limbof the patient or a foot of the patient can be rigidly positioned withinthe positioning device, and the measurement device(s) 102 and/or thecamera(s) 120 track a reference point positioned on the positioningdevice. The processor(s) then can determine the position and movement ofthe bone from the tracked positioning device due to alignment of thelimb of the patient or the foot of the patient within the positioningdevice. With the foot correctly aligned in the positioning device, themeasurement device(s) 102 and/or the camera(s) 120 track the positioningdevice to determine movement. In some examples, it can be difficult totrack movement of the bone or foot itself, and so the positioning devicecan be used to hold the foot rigidly, and tracked to avoid putting pinsin bone to hold in place.

The measurement device(s) 102 and/or the camera(s) 120 also track one ormore reference points positioned on a surgical device, and theprocessor(s) 106 recognizing a position and a movement of the surgicaldevice in physical space based on the tracked reference pointspositioned on the surgical device.

FIG. 5 illustrates an example surgical device 130, according to anexample embodiment. FIG. 6 illustrates another view of the examplesurgical device 130, according to an example embodiment. The surgicaldevice 130 shown in FIG. 5 is a cut guide, and reference points areincluded on a surface of the cut guide that can be tracked by themeasurement device(s) 102 and/or the camera(s) 120. Example referencepoints include reference points 132, 134, and 136. The measurementdevice(s) 102 and/or the camera(s) 120 track the reference points 132,134, and 136 by taking continuous measurements during surgery toidentify movement of the surgical device 130, and provides informationof movement of the surgical device 130 to the processor(s) 106. Theprocessor(s) 106 can update a viewpoint of the surgical device toreflect the position and the movement of the surgical device in physicalspace. In this way, the surgeon can have a visual view of the surgicaldevice 130 on the display 118 that accurately reflects the position ofthe surgical device 130 during surgery.

After all the data above is collected, the measurement device(s) 102and/or the camera(s) 120 will track the motion of the surgical device130 as the surgeon moves the surgical device 130. The processor(s) 106may receive the data and information collected and output by themeasurement device(s) 102 and provide for display a virtual imageoverlaid onto the visual model 122 of the bone. The virtual image may beor include many items, and generally, includes information forassistance of placement of the surgical device 130 at a position on thebone based on the position and the movement of the surgical device 130in physical space.

FIG. 7 is a conceptual illustration of a virtual image overlaid onto avisual model, according to an example embodiment. The visual model inFIG. 7 includes a live image 138 of the bone exposed during surgery, anda virtual image 140 is illustrated overlaid onto the live image 138 ofthe bone showing cut guide positioning. The virtual image 140 includes agraphic of a cut guide and illustrates suggested placement that can becalculated based on many factors, for example, and data collected by themeasurement device(s) 102 and/or the camera(s) 120, such that thegraphic of the cut guide represents a real cut guide 142 as willphysically be in the operating room. Details of placement of the virtualcut guide 140 are further described below. The graphic of the virtualimage 140 may initially be red (or some color to indicate an incorrectposition of the cut guide 142), and the graphic can change colors, suchas to turn green when the surgeon has positioned the cut guide 142 inthe correct position.

Thus, the camera(s) 120 can capture the live image 138 of the boneduring surgery, and the processor(s) 106 can further provide for displaythe virtual image 140 overlaid on the live image 138 of the bone.

Referring back to FIGS. 3 and 4, a cut plane 144 can be determined, andprovided for display as a virtual cutting plane onto the live image 140of the bone. To determine the cut plane, initially, an axis of the limbis determined.

FIG. 8 illustrates a conceptual portion of a limb of a patient as inFIG. 4 with a plane shown, according to an example embodiment. Thereference points 124, 126, and 128 are used to calculate a mechanicalaxis of the limb. Reference points 126 and 128 will be on each side ofthe malleoli, and reference point 124 will be on the anterior cruciateligament (ACL) footprint, as shown in FIG. 4. A midpoint betweenreference points 126 and 128 can be calculated, and then a line 146between the midpoint and the reference point 124 will be the mechanicalaxes for the tibia. From reference point 124 and the mechanical axisline, a plane 148 can be made coincident with reference point 124 andperpendicular to the mechanical axis line 146. Then the cut plane 144can be created by offsetting from the plane 148 by a distance specifiedby a size of the implant selected by the surgeon (10 mm or 11 mm, forexample), as shown in FIG. 3.

Thus, within examples, the processor(s) 106 can determine the axis 146of the limb based on reference points positioned on a medial malleoli ofthe patient, a lateral malleoli of the patient, and a base of ananterior cruciate ligament of the patient, and provide the axis 146 ofthe limb for display. The processor(s) 106 determine a median pointbetween a medial malleoli of the patient and a lateral malleoli of thepatient, and determining a virtual line from the median point up to apoint at the base of the ACL of the patient. The processor(s) 106 canalso calculate the axis 146 of the limb based on the visual model 122and the position and the movement of the bone in physical space, and theaxis can include one of a mechanical axis, a kinematic axis, or ananatomical axis. In some examples, however, implant placement can beassessed without a need to calculate a mechanical axis, such as,assessing implant placement using only information from the registeredbone surface(s).

FIG. 9 illustrates a conceptual portion of a limb of a patient with aplane shown, and FIG. 10 illustrates a conceptual portion of a limb of apatient with a mechanical axis shown, according to an exampleembodiment. In this example, for the femur, point 152 (virtual orphysical) will be placed on the exposed distal femur. The surgeon willrotate the patient's right or left leg and a position of the point 152will be tracked. A plurality of points in addition to point 152 could beplaced physically or virtually on the distal portion of the femur andtracked by the measurement device(s) 102 and/or the camera 120. A centerof rotation can be determined as the point about which the distal femurrotates, generating an arc, a spherical segment, a cap, or a dome 154,and the center point of which is a center of hip rotation noted as point150 in FIG. 10. Then a line 156 from the point at the center of rotationto the placed point(s) will be the mechanical axis, and a plane 144 canbe determined as discussed above. The plane 144 is shown in FIG. 3.

Thus, the axis of the limb can be calculated based on extrapolating apoint of rotation of a hip of the patient during movement of the limbbased on tracked reference points positioned on a distal end of a femur,and the virtual line 156 from the point of rotation of the hip up to apoint at the base of the ACL of the patient can be determined.

As another example, the axis of the limb can be calculated based on dataof an ankle of the patient, and peaks of sides of the ankle can bedetermined from at least two reference points. A median point betweenthe peaks of the ankle is determined, and a virtual line from the medianpoint up to a point at the base of the ACL of the patient is determinedas the axis.

Within examples, because the cut guide 142 has a known cutting planethrough the bone associated with it (referenced from an angle of slot(s)in the cut guide), as the cut guide 142 moves around in reference to thebone, cut plane(s) can be displayed on the screen on the image of theregistered bone. An example cut plane is shown in FIG. 3 as the cutplane 144 that illustrates the projected cut plane generated from thephysical cut guide. The plane 144 moves around in the simulated viewbased on where the surgeon moves the physical cut guide on the patient'sanatomy. This will give the surgeon a virtual view of how the resectionson the bone would look before the surgeon physically makes the cut.Thus, the processor(s) 106 may further provide a virtual view of aresection on the bone for display.

Augmented reality may also be used to overlay the virtual cut planes144, 148, and 158 onto the live image 138 of the patient's exposed femurand tibia. This augmented reality view could also combine aspects of thevisual model 122 of the bone with a live view to generate the bestsuited view for the surgeon.

In some examples, the surgeon is able to virtually place a tibial andfemoral prosthesis intraoperatively on top of the visual model 122 ofthe registered bone. Various views, such as sagittal, coronal,transverse, or isometric views of the modeled bone can be selected. Thevisual model 122 can be manipulated to a custom view to place thevirtual prosthesis.

FIG. 11 illustrates a conceptual portion of the visual model 122 of alimb of a patient with a component(s) of the prosthesis 160 inserted,according to an example embodiment. The prosthesis 160 is shown as agraphic illustrating placement of the prosthesis 160 in relation to thebone.

FIG. 12 illustrates a conceptual portion of the visual model 122 of alimb of a patient with another prosthesis 162 inserted, according to anexample embodiment. In one example, since geometry of the prosthesis 162matches with the plane(s) of the cut guides and all the geometry of thecut guides are known, the processor(s) 106 will calculate where the cutguides will need to be placed in order for the prosthesis 162 to bepositioned where the surgeon virtually placed the prosthesis 162. Themeasurement device(s) 102 and/or the camera(s) 120 will track motion ofthe cut guide in real time using, but not limited to, a tracking patternon the back of each cut guide as described above, to match the cut guideto the known geometry of that specific cut guide. The surgeon my use acut guide positioning apparatus, and the processor(s) 106 illustrateinformation for the surgeon to use to adjust the cut guide into place,so that the cut guides will produce a geometry that match the surgeon'svirtual placement of the prosthesis 162.

FIG. 13 illustrates a conceptual image of placement of a prosthesis 164and additional data to be displayed, according to an example embodiment.Such additional data may include one or more of a lateral gap of a kneeof the patient, a medial gap of the knee of the patient, a varus and/orvalgus alignment of the knee of the patient, a degree of flexion of afemoral cut, a slope of a tibial cut (e.g., tibial plateau cut), and ananteroposterior (AP) rotation extension.

Thus, as shown in FIG. 13, alongside the visual representation,information such as, but not limited to, lateral gap, medial gap, valgusalignment, flexion, and/or AP rotation extension as the informationrelates to the cut being made can be displayed. The measurementdevice(s) 102 and/or the camera(s) 120 will track the motion of the cutguide in real time using, but not limited to, a tracking pattern on theback of the cut guide or using measurement to match the cut guide to theknown geometry of that specific cut guide, and the surgeon may eithermove the cut guide around freely or have the cut guide attached to arefined positioning system that would be attached to the patient. Thiscut guide positioning apparatus would be able to finely adjust the cutguides along the X, Y, and Z axes as well as the pitch, roll, and yaw.The measurement device(s) tracks a position of the guide positioningdevice as well.

Within examples above, any number of combination of information may bedisplayed, and a specific example includes displaying the virtual imageincluding placement of the surgical device as shown in FIG. 7, and agraphic illustrating placement of a prosthesis in relation to the boneand information indicating one or more of a lateral gap, a medial gap,and a valgus alignment of a knee of the patient with placement of theprosthesis, as shown in FIG. 13.

Once the tibial and femoral prosthesis are in place, a position of theprosthesis can be verified with the measurement device(s) 102 and/or thecamera(s) 120 to verify that the prosthesis was placed in the correctposition. The post placement verification will also provide usefulmetrics that will denote success of the surgery, including but notlimited to, lateral gap, medial gap, valgus alignment, flexion, or AProtation extension. Thus, the measurement device(s) 102 and/or thecamera(s) 120 may further scan or measure a prosthesis after placementin the bone and output prosthesis data, and the processor(s) 106verifies placement of the prosthesis in the bone to a determinedposition based on the prosthesis data. The method can be repeated inpart or in whole if the prosthesis is not positioned correctly accordingto the system or based on the judgement of the surgeon.

FIG. 14 shows a flowchart of an example method 200 for determining andproviding for display a virtual image overlaid onto a visual image of abone, according to an example embodiment. Method 200 shown in FIG. 14presents an embodiment of a method that could be used with the system100 shown in FIG. 1 and the computing device 104 shown in FIG. 2, forexample. Further, devices or systems may be used or configured toperform logical functions presented in FIG. 14. In some instances,components of the devices and/or systems may be configured to performthe functions such that the components are actually configured andstructured (with hardware and/or software) to enable such performance.In other examples, components of the devices and/or systems may bearranged to be adapted to, capable of, or suited for performing thefunctions, such as when operated in a specific manner. Method 200 mayinclude one or more operations, functions, or actions as illustrated byone or more of blocks 202-210. Although the blocks are illustrated in asequential order, these blocks may also be performed in parallel, and/orin a different order than those described herein. Also, the variousblocks may be combined into fewer blocks, divided into additionalblocks, and/or removed based upon the desired implementation.

It should be understood that for this and other processes and methodsdisclosed herein, flowcharts show functionality and operation of onepossible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer readable medium ordata storage, for example, such as a storage device including a disk orhard drive. Further, the program code can be encoded on acomputer-readable storage media in a machine-readable format, or onother non-transitory media or articles of manufacture. The computerreadable medium may include non-transitory computer readable medium ormemory, for example, such as computer-readable media that stores datafor short periods of time like register memory, processor cache andRandom Access Memory (RAM). The computer readable medium may alsoinclude non-transitory media, such as secondary or persistent long termstorage, like read only memory (ROM), optical or magnetic disks,compact-disc read only memory (CD-ROM), for example. The computerreadable media may also be any other volatile or non-volatile storagesystems. The computer readable medium may be considered a tangiblecomputer readable storage medium, for example.

In addition, each block in FIG. 14 may represent circuitry that is wiredto perform the specific logical functions in the process. Alternativeimplementations are included within the scope of the example embodimentsof the present disclosure in which functions may be executed out oforder from that shown or discussed, including substantially concurrentor in reverse order, depending on the functionality involved, as wouldbe understood by those reasonably skilled in the art.

At block 202, the method 200 includes receiving information, collectedby the measurement device(s) 102 and/or the camera(s) 120, of a surfaceof a bone of a limb of a patient.

At block 204, the method 200 includes determining the visual model 122of the bone based on the collected information of the surface of thebone.

At block 206, the method 200 includes recognizing a position and amovement of the bone in physical space based on tracked reference pointspositioned on the one or more of the bone or the anatomy of the patientand causing the visual model of the bone to reflect the position and themovement of the bone in physical space. Position coordinates of thetracked reference points can be determined referenced to the measurementdevice(s) 102 and/or the camera(s) 120 performing the tracking.

At block 208, the method 200 includes recognizing a position and amovement of a surgical device in physical space based on trackedreference points positioned on the surgical device.

At block 210, the method 200 includes providing for display a virtualimage overlaid onto the visual model of the bone, and the virtual imageincludes information for assistance of placement of the surgical deviceat a position on the bone based on the position and the movement of thesurgical device in physical space.

The method 200 may be performed in real time during surgery to collectthe information of the surface of the bone as exposed from the limb ofthe patient. For example, the information can be received usingmeasurement device(s) 102 and/or the camera(s) 120 in real time duringsurgery.

As described within examples above with respect to the system 100, themethod 200 can include providing a virtual view of a resection on thebone, and can display the virtual image including placement of thesurgical device, a graphic illustrating placement of a prosthesis inrelation to the bone, and information indicating one or more of alateral gap, a medial gap, and a valgus alignment of a knee of thepatient with placement of the prosthesis.

In addition, as described within examples above with respect to thesystem 100, the method 200 can include calculating an axis of the limbbased on the visual model and the position and the movement of the bonein physical space, and the axis is one of a mechanical axis, a kinematicaxis, or an anatomical axis.

In an example method, virtual or physical points can be placed on themedial and lateral malleoli of the patient, along with the base of theACL foot print of the tibia. The method 200 can be further executed tocreate a plane in space that is referenced off of the three referencepoints (as described above with respect to FIGS. 3, 4, and 8). Thismethod will then show surgeons the mechanical axis plane of the tibiafrom the sagittal view. To create the mechanical axis line, the mediandistance between the two malleoli's can be interpreted, and a virtualline from the median line up to the point that sits at the ACL footprint can be determined. Once the mechanical axis is established, thecutting guide will be able to be adjusted in multiple different axes tocreate the proper resection, such as but not limited to, the posteriorslope, height, and rotation. This can be accomplished by either having asupportive fixture that allows for adjustments of the cutting guideitself, or by having cut guides that contain mechanisms that allow forfurther adjustments once they have been pinned to the bone. Themeasurement device(s) 102 and/or the camera(s) 120 will be able to trackthe movement of the cutting guide by having reference points attacheddirectly on the cutting guide, or by having a temporary insert thatengage with the cut blocks with reference points. Once the correctposition is met by image recognition (due to tracked reference points bythe measurement device(s) 102 and/or the camera(s) 120), the surgeon canmake the tibial plateau cut. In another instance, once the scan of theexposed tibia and the lower portion of the knee is acquired, theprocessor(s) 106 can calculate the peaks of both sides of the ankle fromat least two planes, the sagittal and coronal, but not limited to. Thiscan be an alternative method to calculate the midline between themedical and lateral malleoli, and have the processor(s) 106 draw avirtual line that connects the virtual midline from the patient's ankleup to the ACL footprint.

To create a sagittal cut or vertical cut for the tibial resections,additional instruments could be utilized to give surgeons a numericalvalue tied to dimensions of the tibia. In an example method, a probe canbe used where one end has a hook or curve. The hooked end would engagewith the posterior of the tibia and the system 100 can recognize theother end of the probe in relation to a metal cutting instrument or anyother reference point to give the surgeon a numerical value of the totallength of the tibial plateau.

In another example method for the sagittal cuts, the surgeon can alsoutilize a dual pronged probe that is of a certain fixed width. Thesurgeon would insert the fork into the patient's knee above the tibialplateau. The measurement device(s) 102 and/or the camera(s) 120 wouldrecognize this specific instrument, and when the ends of the fork are inthe midline, or a set distance in reference to the virtual point orphysical point of the ACL footprint and the lateral or medial edge ofthe tibia, the measurement device(s) 102 and/or the camera(s) 120 wouldnotify the surgeon and a measured sagittal resection can be made. Thisdevice can either be a standalone device, or be engaged or connected tothe tibial cutting guide or any other fixed instrument.

In another example method, the exposed distal condyle of the femur isregistered through the measurement device(s) 102 and/or the camera(s)120, and the majority of the leg is also registered. A plurality ofvirtual or physical reference points can be placed on the femoralcondyles, and used to calculate the center of the condyles. As thesurgeon moves the hip the system 100 extrapolates the point of rotationby tracking the reference points on the distal end of the femur. Oncethe point of rotation is calculated, a virtual line can be created,which will be the mechanical axis, portions of which are shown anddescribed above with reference to FIGS. 3 and 9-10, for example.

Example methods and systems described herein provide visual feedback toassist the surgeon in the placement of the cut guide or the prosthesis,and calculate measurement data to assist the surgery and provide realtime data. The tracking of the bone and surgical devices with thespatial awareness to the bone to calculate the mechanical axes of theleg (without specifically identifying the endpoints of such mechanicalaxes, such as with a probe) is provided to assist the surgeon in placingthe cut guides in the correct position onto the bone. Any number orarray of measuring devices can be used to provide real time data as wellas object tracking when the surgeon moves the cut guides, and the system100 can further include augmented reality with the real time measurementarray to assist the surgeon to place the cut guides by displaying thevirtual cutting planes onto the live image of the bone in reference tothe physical cut guide before the cut guide is fixed into place.

Example methods and systems described herein can be used during surgery,and generally, during any surgery, as a safety mechanism to preventvariation from a desired result. Example methods and systems describedherein can manage variables and inform surgeons to place prostheseswithin a certain threshold of accuracy. Some example surgeries for useof the methods and systems include unicompartmental knee arthroplasty(UKA), total knee arthroplasty (TKA), total hip arthroplasty (THA), andresurfacing knee and hip arthroplasties.

FIG. 15 shows a flowchart of another example method 220 for positioningof a prosthesis, according to an example embodiment. Initially, as shownat block 222, a scan of the bone or anatomy of the patient is performed(e.g., by the measurement device(s) 102). As shown at block 224, theknee is scanned as thoroughly as possible when the knee is exposed, forexample.

Next, as shown at block 226, cameras will capture images and overlay thescanned bone onto the images of the actual bone, and as shown at block228, when the knee moves (such as repositioning during surgery), thescan data will move with the knee. The augmented reality data includes alive direct or indirect view of a physical, real-world environment ofthe surgery whose elements are augmented (or supplemented) bycomputer-generated sensory input such as sound, video, or graphics data.In this example, the augmented reality will give the surgeon a real timeview of the surgery with scanned data overlaid to help the surgeonprecisely place the cut guides. The system may provide an augmentedreality display including the live image of the bone in physical spacesupplemented by a computer-generated graphic of a cutting plane overlaidon the bone, for example.

Following, as shown at block 230, the surgeon will place the implant(tibial then femoral) virtually using the system to best match a naturalgeometry of the knee. As shown at block 232, different views can beutilized on the software to place the implant virtually, and as shown atblock 234, once the implant is correctly positioned (within thresholdsset for the surgery), the system can display information for a remainderof the surgery.

As shown at block 236, the system can calculate where all cutting guidesneed to be placed in reference to the scan data. At block 238, interiorgeometry of the implant is used in reference with planes in which thecut guide uses to calculate positioning of the cut guides.

At block 240, the surgeon positions the tibial cut guide in reference tothe indicated positioned by the system. At block 242, the surgeonpositions the distal femoral cutting guide in reference to the indicatedposition by the system. At block 244, the distal cut is shown to beequal to the posterior cut, which is equal to the flexion gap minus thetibial cut. At block 246, the surgeon positions the posterior andchamfer cutting guide in reference to the indicated position by thesystem. At block 248, the system projects an image of the cut guides onthe screen, and the surgeon references this while placing the cut guidesuntil the images on the screen change to green and the cut guides arepinned in place indicating correct positioning of the cut guides.

At block 250, the cut planes can be displayed by the system on the scandata after the cut guide is placed.

Following, at block 252, the surgeon places the cemented tibial andfemoral component in place. At block 254, the positioning of thecomponents is checked to verify correct placement. This may beaccomplished by scanning the inserted components to match against theposition indicated by the system, for example.

FIG. 16 shows a flowchart of another example method 260 for positioningof a prosthesis, according to an example embodiment. Initially, as shownat block 262, a scan of the bone or anatomy of the patient is performed(e.g., by the measurement device(s) 102).

As shown at block 264, the mechanical axes of the leg are calculated bythe system through motion tracking. As shown at block 266, the surgeonwill then place the tibial positioning guide. Following, at block 268,the tibial cut guide will be placed in the positioning system. Next, atblock 270, the surgeon will place the femoral cut guide positioningsystem. At block 272, the surgeon will place the distal cut guide in thepositioning system. At block 274, the posterior and chamfer cut guidewill be positioned in the same way.

At block 276, the cameras will recognize placement of the cut guides,and the system will display the prosthesis on the scan data in referenceto the cut guide positioning so that the surgeon can perform fine tunepositioning. In this way, the placement of the prosthesis occursvirtually to allow the surgeon to view the placement prior to making anycuts on the patient.

At block 278, the surgeon can also change a view to display planes ofthe cut virtually projected onto the scan data. At block 280, the systemmay display numerical values, such as the valgus alignment and otherdata in real time.

Following, at block 282, the surgeon places the tibial and femoralcomponents in place, and at block 284, positioning of the components isverified.

Example methods and systems thus provide live real time images withvirtual guides placed to show cutting planes and/or prosthesis positionsin real time, prior to making any cuts. In the operating room, cutguides can be positioned on the bone in real time, and simultaneously,the system can display cut planes and prosthesis positions virtually tovisualize the reconstruction in real time. This provides a demonstrationto the surgeon of where the prosthesis is to be positioned. The bone andsurgical instruments have reference points that are tracked by themeasurement device and/or cameras to provide a real time up-to-dateimage and virtual image of the cut guides positioning, as well asprosthesis alignment.

The description of the different advantageous arrangements has beenpresented for purposes of illustration and description, and is notintended to be exhaustive or limited to the embodiments in the formdisclosed. Many modifications and variations will be apparent to thoseof ordinary skill in the art. Further, different advantageousembodiments may describe different advantages as compared to otheradvantageous embodiments. The embodiment or embodiments selected arechosen and described in order to best explain the principles of theembodiments, the practical application, and to enable others of ordinaryskill in the art to understand the disclosure for various embodimentswith various modifications as are suited to the particular usecontemplated.

What is claimed is:
 1. A system comprising: a measurement device tocollect information of a surface of a bone of a limb of a patient, totrack one or more reference points positioned on one or more bones or ananatomy of the patient, and to track one or more reference pointspositioned on a surgical device; one or more processors; and datastorage including instructions executable by the one or more processorsfor performing functions comprising: determining a visual model of thebone based on the collected information of the surface of the bone;recognizing a position and a movement of the bone in physical spacebased on the tracked reference points positioned on the one or more ofthe bone or the anatomy of the patient and causing the visual model ofthe bone to reflect the position and the movement of the bone inphysical space; recognizing a position and a movement of the surgicaldevice in physical space based on the tracked reference pointspositioned on the surgical device; and providing for display a virtualimage overlaid onto the visual model of the bone, wherein the virtualimage includes information for assisting in placement of the surgicaldevice at a position on the bone based on the position and the movementof the surgical device in physical space.
 2. The system of claim 1,wherein the one or more processors perform the functionsintraoperatively during surgery.
 3. The system of claim 1, wherein thefunction of providing for display the virtual image comprises: receivinga live image of the bone; providing for display the live image of thebone; and displaying a virtual cutting plane onto the live image of thebone.
 4. The system of claim 1, wherein the one or more processorsfurther perform functions comprising: providing a virtual view of aresection on the bone.
 5. The system of claim 1, wherein the function ofproviding for display the virtual image further comprises: providing fordisplay a graphic illustrating placement of a prosthesis in relation tothe bone.
 6. The system of claim 1, wherein the function of providingfor display the virtual image further comprises: providing for displayinformation indicating one or more of a lateral gap of a knee of thepatient, a medial gap of the knee of the patient, a valgus alignment ofthe knee of the patient, a flexion of a femoral cut, and a slope of atibial cut.
 7. The system of claim 1, wherein the function of providingfor display the virtual image comprises: displaying the virtual imageincluding placement of the surgical device, a graphic illustratingplacement of a prosthesis in relation to the bone, and informationindicating one or more of a lateral gap, a medial gap, and a valgusalignment of a knee of the patient with placement of the prosthesis. 8.The system of claim 1, further comprising: one or more cameras forcapturing a live image of the bone; and wherein the function ofproviding for display the virtual image comprises further providing fordisplay the virtual image overlaid on the live image of the bone.
 9. Thesystem of claim 1, further comprising: wherein the measurement device isone or more cameras for capturing live images.
 10. A method comprising:receiving information, collected by a measurement device, of a surfaceof a bone of a limb of a patient; determining a visual model of the bonebased on the collected information of the surface of the bone;recognizing a position and a movement of the bone in physical spacebased on tracked reference points positioned on one or more of the boneor an anatomy of the patient and causing the visual model of the bone toreflect the position and the movement of the bone in physical space;recognizing a position and a movement of a surgical device in physicalspace based on tracked reference points positioned on the surgicaldevice; and providing for display a virtual image overlaid onto thevisual model of the bone, wherein the virtual image includes informationfor assisting in placement of the surgical device at a position on thebone based on the position and the movement of the surgical device inphysical space.
 11. The method of claim 10, further comprising:receiving the information of the surface of the bone as exposed from thelimb of the patient in real time during surgery.
 12. The method of claim10, wherein receiving the information, collected by the measurementdevice, of the surface of the bone of the limb of the patient comprises:receiving the information using a scanning tool in real time duringsurgery.
 13. The method of claim 10, further comprising: providing avirtual view of a resection on the bone.
 14. The method of claim 10,further comprising: displaying the virtual image including placement ofthe surgical device, a graphic illustrating placement of a prosthesis inrelation to the bone, and information indicating one or more of alateral gap, a medial gap, and a valgus alignment of a knee of thepatient with placement of the prosthesis.
 15. A non-transitorycomputer-readable medium having stored thereon instructions that whenexecuted by a computing device that includes one or more processorscauses the computing device to perform functions comprising: receivinginformation, collected by a measurement device, of a surface of a boneof a limb of a patient; determining a visual model of the bone based onthe collected information of the surface of the bone; recognizing aposition and a movement of the bone in physical space based on trackedreference points positioned on one or more of the bone or an anatomy ofthe patient and causing the visual model of the bone to reflect theposition and the movement of the bone in physical space; recognizing aposition and a movement of a surgical device in physical space based ontracked reference points positioned on the surgical device; andproviding for display a virtual image overlaid onto the visual model ofthe bone, wherein the virtual image includes information for assistingin placement of the surgical device at a position on the bone based onthe position and the movement of the surgical device in physical space.16. The non-transitory computer-readable medium of claim 15, wherein thefunction of providing for display the virtual image comprises: receivinga live image of the bone; providing for display the live image of thebone; and displaying a virtual cutting plane onto the live image of thebone.
 17. The non-transitory computer-readable medium of claim 15,wherein the function of providing for display the virtual imagecomprises: receiving the information of the bone as exposed from thelimb of the patient in real time during surgery.
 18. The non-transitorycomputer-readable medium of claim 15, wherein the function of receivingthe information, collected by the measurement device, of the surface ofthe bone of the limb of the patient comprises: receiving the informationusing a scanning tool in real time during surgery.
 19. Thenon-transitory computer-readable medium of claim 15, wherein thefunction of providing for display the virtual image comprises: providinga virtual view of a resection on the bone.
 20. The non-transitorycomputer-readable medium of claim 15, wherein the function of providingfor display the virtual image comprises: displaying the virtual imageincluding placement of the surgical device, a graphic illustratingplacement of a prosthesis in relation to the bone, and informationindicating one or more of a lateral gap, a medial gap, and a valgusalignment of a knee of the patient with placement of the prosthesis.